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Introduction 

The process of development and improvement requires constant evaluation and 
change. This is true of a person or an educational program. The evaluation element is 
critical to determine which parts of the program are worthy of continuation; which need to 
be modified to better fit the circumstances; and which need to be discarded/replaced. The 
result is a spiraling effect of application of a treatment and evaluation of its effectiveness 
which brings the person or the agency closer to the ultimate goal. 

In focusing specifically on the evaluation portion of this process, one needs to 
develop an evaluation strategy that is situationally appropriate. A theoretical framework 
must be interwoven into the structure of the evaluation (Figure 1). The evaluator must 
analyze the evaluation work that has been completed previously to determine what is still 
needed. Finally, the application of the background and chosen design to the situation. This 
writer has chosen to evaluate the relevance of long term U.S. graduate education for 
students from Malawi, East Africa. 

During the past decade, the developing countries of the world have invested more 
than two billion dollars in training of nationals. Training is aimed at upgrading skills and 
building organizational capacity through in-country and long-term, off-shore training. 
Richard Clough, Regional World Bank Agricultural Training Specialist for the African Desk, 
estimates that World Bank, USAID, and other donors have invested more than $50 million 
in manpower development since 1980. Major donors have invested in Malawi, most of 
which was in the area of long-term, off-shore training at the masters and doctorate levels. 
Some 75 officers from the Ministry of Agriculture have been sponsored for graduate 
education by USAID since 1980. 
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Figure 1. 




Officials from Malawi and USAID are vitally interested in the impact of this training. 
A central question pertains to the relevancy of the training to both the individual and the 
agricultural development of Malawi. The USAID position is well articulated by the Mission 
Director Carol Pensley in 1989 when she said, "It is essentia, to determine both 
quantitatively and qualitatively the impact of the Mission's long term commitment to masters 
and doctorate training of Ministry of Agriculture officers in the U.S. One of the first steps 
is to view the training in terms of relevancy and impact of the agricultural development of 
Malawi." 

An evaluation process was established to determine the effectiveness of long-term 
graduate education with Malawi students. The writer's analysis of this evaluation will be 
broken down into three main areas: 

• The theoretical basis for the evaluation; 

• Critique of the Malawi evaluation within that framework; 

• Construction and application of evaluation techniques to a pre-departure 
orientation program. 

The theoretical framework includes the selection of the evaluation design given the 
constraints of the situation and guarding against threats to the internal and external validity 
of the study. An evaluation model will be selected and justification for its use given. The 
research techniques used for data collection will also be explored. In addition to the text 
used in the evaluation class, the writer will incorporate insight into the evaluation process 
from outside sources. 

After establishing the theoretical basis for the evaluation, the evaluation conducted 
for the Government of Malawi (GOM) by the Department of Adult and Youth Education 
at Washington State University will be analyzed. This will include information on the 
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research procedures, design and constraints. Information for this section was obtained 
through written reports and interviews with program evaluators. The analysis will be 
structured around the theoretical basis which has been developed. Included in the analysis 
will be a critique of the insti-ument used and a reaction to the information obtained. 

The final section will report the application of the evaluation model and research 
des ; gn selected to a portion of the Malawian evaluation. A pre-departure orientation 
program conducted for students scheduled to begin long-term graduate study in the United 
States will be the focus of the evaluation. Included in this section will be the educational 
outcomes to be evaluated, instrument chosen to evaluate the outcomes, administration 
procedures, results and a discussion of the limitations and conclusions drawn from this 
evaluation procedure. 
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Theory 

The usefulness of information gained from an evaluation can depend largely on the 
care that is taken by the evaluator when planniag the evaluation. The evaluation design and 
evaluation model must be carefully selected while keeping in mind the specifics of the 
program. There are three broad categories of evaluation designs: non-experimental, quasi- 
experimental, and true experimental. Within each of these broad categories, there are 
several types of evaluation designs. The elements used to distinguish one type from another 
include random vs. non-random selection, centralized or decentralized selection for the 
treatment, administration of a pre-test, presence of a control or comparison group and the 
timing of the assessment/treatment process. Selection is the "locus of decision regarding 
which treatment a given subject will receive." (Mohr, 1988 p.45) This selection process can 
determine which of the evaluation designs is the most appropriate for the evaluation. 
Random selection designates that the participants are being assigned to treatment and to 
comparison or control groups at random. Each individual has an equal chance of being in 
either group, whereas in non-random selection this is not the case. Centralized selection 
states that assignment to the groups is being made by one individual or group of individuals. 
The same criteria for assignment is being used on each individual. Testing the participants 
prior to the treatment to establish a baseline for interpretation of any differences produced 
by the treatment is another facet of determining which evaluation design to be used. Non- 
experimental designs do not utilize pre-testing procedures. 

In selecting a specific design, the evaluator is concerned with two main things: 
external and internal validity. Internal validity is the capability of an evaluative research 
study to establish that the observed program outcomes were actually produced by the 
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program operations rather than extraneous influences. External validity is the degree to 
which the research findings may be generalized to a larger population (Miller, 1979). Given 
the circumstances of the program to be evaluated, the researcher must select the design that 
guards against as many threats to validity as possible. Threats to validity are alternative 
elements that could have brought about the change in the program participants. These 
threats include history, maturation, test-wiseness, attrition, selection, contamination and 
spuriousness. The evaluation designs can be placed on a continuum according to how well 
they counter those threats. This continuum would proceed from non-experimental to 
experimental with experimental being the most effective at guarding against the threats, 
non- quasi - 

experimental experimental -* experimental 

Experimental designs however do not often occur in real life and therefore they are the 
most difficult to relate to real life situations. They are often more easily constructed, but 
the circumstances surrounding the experiment can be rather contrived and therefore less 
applicable to the world outside the experiment. Since educational programs are not often 
constructed to meet all the criteria of experimental designs, evaluators must choose alternate 
methods that fit the situation. In choosing those alternate methods, the evaluator must be 
aware of the threats to validity inherent with that design and the limitations posed on the 
interpretation una use of the results. 

One of the designs categorized in the non-experimental group is the ex-post-facto or 
"after the fact" design. This design is quite low on the continuum and therefore there are 
several threats to validity that the evaluator must factor in. There are two elements that 
characterize the ex-post-facto design: 

• reconstruction of events 

erJc 1 U 



• decentralized selection of sample. 

Ex-post-facto studies are categorized at a lower level on the research design continuum 

partly because of these two factors. There is a time lapse between the administration of the 

treatment and the assessment. Therefore, the participants must reconstruct the events that 

took place in order to evaluate the effectiveness of the treatment. 

Perhaps more important than reconstruction is the decentralized selection into the 

treatments. This is the main factor that characterizes ex-post-facto designs. Subjects are 

not assigned to the treatment by one person or group. According to Mohr, 

"An ex-post-facto design is one in which no central authority decides which subjects 
are to receive the treatment and which are not, nor what intensity of treatment each 
is to receive. Rather, that decision is made by different persons for different 
subjects." (Mohr, 1988 p.166) 

Decentralized selection was a large part of the Malawi evaluation. The participants were 
selected and sponsored by different agencies. They were placed by those sponsors in 
different universities and participated in different graduate programs for varying lengths of 
time. Hence this evaluation design has been selected by this writer for evaluating the 
effectiveness of the long-term, off-shore graduate education. 

This decentralized selection is also responsible for the largest threat to validity of this 
type of evaluation design, spuriousness. An evaluator attempts to establish whether the 
treatment is producing the outcome of interest. In order to make that determination, the 
evaluator tries to reduce or account for the effect of extraneous factors on the outcome. 
Spuriousness is when a variation in some factor (Z) causes both the treatment (T) and the 
outcome (Y) to vary in such a way as to be observationally related to each other, but not 
due to one causing the other (Mohr, 1988). The ex-post-facto design implies a certain pre- 
test value that is unmeasured and this can be a major source of spuriousness. In the case 



n 



8 

of the Malawi study, there was no pre-test value to determine a baseline for the evaluation 
and the sources of spuriousness are unlimited. 

Another threat to validity in the ex-post-facto design is that of time-order sequencing. 
In the reconstruction, it is difficult to establish the exact order of key events. This order 
needs to be established to determine if the treatment is producing the observed outcome. 
The time-ordering in the case of Malawi was reconstructed by the evaluators in conjunction 
with GOM officials. The evaluators felt secure with the sequence established, however, this 
potential threat must be acknowledged. 

After selecting an evaluation design that fits the program environment and purpose, 
the evaluator must determine the evaluation model to be used. In a book titled: A Source 
Book for Program Evaluation and Accountability, Phyllis Worden outlines several different 
evaluation models and their uses. Robert W. Miller also suggests an evaluation model used 
for rural development evaluation (Miller, 1979). Miller's Program Outcome model will be 
the model applied to the evaluation of the orientation program, therefore it will be the focus 
of this section. Given the constraints of the ex-post-facto design and the situation of the 
Malawi evaluation, Miller's model is well suited for this evaluation. 

The Program Outcome Model suggests breaking the educational program into a 
succession of program procedures and outcomes which will need to occur if the program is 
to reach its ultimate goal. This model of outlining the anticipated flow of operations and 
outcomes is a useful guide in developing the major elements of the evaluation strategy. 
Each program is divided into short, intermediate, and long-term outcomes. These outcomes 
are presented in a chain in which short-run outcomes lead to intermediate outcomes which 
contribute finally to the long-run goal. These outcomes may be program procedures as well 
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as outcomes, hence this model can be used at various stages in the planning, implementation 
and evaluation of a program to determine the present status of the program. In addition, 
an outcome line, as suggested in Mohr's book, can be developed for each of the outcomes 
in the Program Outcome Model. This approach is beneficial when there is a large problem 
to be evaluated because it breaks the large evaluation task into smaller evaluation projects 
over which the evaluator has more control of intervening variables, those variables which 
could account for the outcome that are not part of the treatment. 

In the specific case of the Malawi evaluation, the ultimate outcome of agricultural 
development is separated into predecessor outcome's. Each of these outcomes has an 
evaluation element as part of its design in order to determine if the outcome has been 
achieved and its effectiveness (Figure 2). This evaluation will concentrate on the training 
outcome. The Malawi evaluation conducted by AYE in the summer of 1990 incorporated 
an assessment of each of the training elements: long-term, off-shore graduate education, 
short-term, off-shore training, and training of trainers. A Mohr Outcome Line has been 
developed for the lon t ' T term, off-shore graduate training outcome (Figure 3). Outcome lines 
visually display the outcome of interest broken down into its component sub-objectives and 
the activities that contribute to each of those sub-objectives. By breaking down each of the 
outcomes in Miller's Program Outcome Model further using the Outcome Line design 
suggested by Mohr, the evaluator can define each step of the educational program and 
specific tasks that may be more readily evaluated. This process also helps the evaluator to 
establish the relationships between the sub-objectives and the ultimate goal. The outcome 
line for th Malawi study will be addressed in detail in the Critique section of this paper. 
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Figure 2. 
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Figure 3. 
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The program outcome model was constructed to address the issues concerning rural 
development. It is designed for programs that take a long time to reach their ultimate goal. 
This model would be quite cumbersome for a short term program and would probably not 
be necessary. Miller indicates that by separating the evaluation task into outcomes and 
evaluating those outcomes it is easier to identify which parts of the program achieved their 
desired aim and which did not. He cautions that the lack of evidence that the program 
achieved its objective may be due to the sensitivity of the instrument used to assess the 
outcome. Sensitivity of the instrument is an issue that will be discussed further in the 
Critique and Implementation sections. 

In building the theoretical basis for the analysis of the Malawi evaluation there are 
two other non-experimental research designs that should be explained. These designs are: 

• Case Studies 

• Site Visits 

Case studies use a wide range of available information about a program to build a narrative 
description of program processes and outcomes (Miller, 1979). This information is gathered 
using techniques which include: informal conversations with program participants, analysis 
of program reports, systematic surveys of program participants and staff members. The case 
study produces a hodge podge of different information which may constitute a pattern of 
meaningful and convincing findings (Miller, 1979). One of the main problems of this type 
of evaluation is that this design lacks the type of in-depth information that is produced by 
experimental and quasi-experimental research. At best, it produces a shot-gun approach to 
evaluation. There is also the threat of bias by the evaluator in the selection of which 
information should be included and the interpretation of that information. However this 
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type of research often best resembles real life situations. Caution should be taken when 
ascribing the outcome as a direct result of the treatment. Many aspects of the Malawi 
evaluation resemble some of the factors included in the case study research design and it 
was therefore important to introduce this non-experimental design. The same is true for the 
site visit method. 

Site visit evaluation refers to a limited assessment of a program made by an evaluator 
during the course of a rather brief visit to a program site. Aspects of the program operation 
are selected for study. This study includes interviews with selected staff members and 
program participants, as well as document analysis. The evaluator's own standards can play 
an important role in the judgements that ar made (Miller, 1979). When using this type of 
evaluation, these shortcon. v must be acknowledged. This method, though flawed, can 
often be better than no evaluation at all. 

A final theoretical area that should be addressed in order to fully understand the 
procedures used in the Malawi evaluation is that of inside vs. outside evaluation. In the 
case of the Malawi project, the Government of Malawi commissioned a group of outside 
evaluators to conduct the evaluation of their training procedures. According to Miller, 
several factors enter into a decision to use outside evaluators. The threat felt by members 
of the organization will be increased when using outside evaluators. Time will be needed 
for the outside evaluator to get acquainted with the program. Complexities and nuances 
may be overlooked by an outside evaluator (Miller, 1979). Conversely, the outside evaluator 
can give a more objective assessment of the program. The objectivity of the evaluation is 
affected by the stake that the individual has in the success of the project or outcome. An 
inside evaluator can be biased by his/her interest in seeing the program succeed or fail. On 
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the other hand, outside evaluators judge the program based on measured outcomes and may 
not fully appreciate the context of the program. 

It is important to establish the theoretical framework within which the evaluation will 
be conducted in order to understand the limitations of the evaluation itself and the 
application of the results. Since evaluation is the estimation of the "value" of a program or 
outcome, it is vital to select the design that fits the needs of the program and maximizes the 
use of the evaluation findings. 
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Critique 

With this theoretical background, the next step will be to apply this information to 
a critique of the evaluation conducted for the GOM in the summer of 1990. This critique 
includes information th? i obtained through interviews with the evaluators, analysis of 
the instruments used, and reports submitted to the Government of Malawi and USD A. It 
will include background information on the evaluation, an analysis of the data gathering 
techniques and instruments used to assess program effectiveness, and the possible uses of 
the information. 

In December 1989 approval was given from the Malawian Ministry of Agriculture to 
conduct an evaluation of the training component of the National Rural Development 
Programme V (NRDP V). This evaluation was a midway assessment of the effectiveness 
of the training element of the Government of Malawi's Ten Year Development Plan. The 
Ministry of Agriculture, with financial support from World Bank, commissioned the USDA 
for the evaluation. USDA awarded the evaluation contract to the Department of Adult and 
Youth Education at Washington State University. Two evaluators, Dr. Ron Jimmerson and 
Dr. Jim Long, from the department conducted the evaluation. They were assisted by Dr. 
R. Mkandawire and several students from Bunda College. Also a small committee from the 
Ministry of Agriculture was to have been formed to work with the evaluators. Dr. Tom 
Trail was serving as a Training Advisor in the USAID Mission in Malawi and contributed 
to this evaluation. 

Dr. Ron Jimmerson traveled to Malawi in April of 1990 to begin the evaluation 
process. The evaluation of training included three main types of training: 
• Long-term, Off-shore Graduate Education 
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• Short-term, Off-shore Training 

• Training of Trainers. 

The objectives of the evaluation were outlined before the evaluator traveled to Malawi. The 
first two weeks were spent interviewing various stakeholders in the training and evaluation 
process. These interviews were to collect data related to the identification of expectations 
for the evaluation and to identify issues and concerns related to training (Jimmerson, 1990). 
Based on those interviews and document analysis, questionnaires were developed to collect 
data from those who had participated in the training. Two types of questionnaires were 
administered to the three groups of individuals. One questionnaire was administered to 
those who participated in short-term, off-shore training and the in-country Training of 
Trainers. These two groups were compared to each other (Appendix A). Participants in 
long-term, off-shore graduate education were surveyed using a questionnaire developed by 
David Acker and used in a study of long-term training graduates in Tanzania (Appendix B). 
An additional section was added to the Acker survey to assess specific skill gains in the 
areas of task-oriented and process-oriented skills. This section was developed by Dr. 
Jimmerson. Dr. Trail was instrumental in locating the Acker survey and assisting with data 
collection. 

The data compilation and analysis were conducted under the direction of Dr. 
Mkandawire and other faculty members at Bunda College. In July of 1990, Dr. Jim Long 
traveled to Malawi to interpret the data and to write the initial report for the Government 
of Malawi. Upon return to the U.S., the final report of the data was written and distributed 
in November, 1990. 

The evaluators utilized several methods for data collection. As stated earlier, Dr. 
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Jimmerson interviewed stakeholders. He noted that the interviews were generally 
unstructured and informal. Through this process, he was able to make the individual more 
at ease and get valuable information about the training process and elements that should 
be involved in the evaluation. As an outside evaluator, he had some preconceptions about 
the state of the training in Malawi, but he found those to be false. In fact, he was quite 
surprised with the complexity of their training procedures. He indicated that there were 
very few problems with language when he was interviewing the government officials and 
faculty members. He did utilize an interpreter when he proceeded into the field to talk with 
the local agriculturists. Jimmerson also spent time analyzing government reports on training 
to gain some perspective on the history of the training program. In the theory section of 
this report, the elements of case study and site visits were explained. This study 
incorporated elements of those non-experimental designs. It was also conducted by an 
outside evaluator. That could have affected the findings of the evaluation. 

For the purposes of this critique, the long-term, off-shore graduate education element 
of this evaluation will be evaluated to the exclusion of the others. This element is one that 
U.S. universities, faculty, and administrators have the most control over and subsequently 
can make the most difference. The evaluation design utilized in the Malawi evaluation is 
definitely ex-post-facto. There was not pre-test to establish baseline data. The evaluators 
and the participants are charged with reconstructing the program events of the previous five 
years. The selection of the individuals to treatments was decentralized; the report on by 
Long, et al 1990 indicated that the participants were selected and sponsored by different 
agencies for graduate study in the U.S. The level of the treatment may have been different 
for each participant given the fact that many participants attended different universities, 
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pursued different degree programs, and encountered different cultural circumstances. 

An instrument was developed by David Acker to assess the perceived relevance of 
graduate education to the participant's career and their country's agricultural development 
needs. This instrument included 28 questions that dealt with the relevance of education to 
career development. Each question solicited information in two sections: one that addressed 
the emphasis in training and the relevance to career. The items were rated on a Likert 
scale (1 = very relevant, 5 =not relevant). There were also 9 questions pertaining to national 
agricultural development arranged in the same fashion. The third section gathered 
demographic information. In addition, another section assessed the increases in skills as a 
result of graduate education. There was no information available on the validity or 
reliability of the instrument. The questionnaire collected qualitative data on the 
participants' perceptions of relevance, not quantitative data on actual application of the 
graduate education to their careers and national agricultural development. 

Phyllis Worden outlined some advantages and disadvantages of using questionnaires. 
Some of the advantages to using questionnaires are that they are easily coded and they are 
easily adapted to sampling techniques, useful with large numbers of people in differing 
geographical regions, they can be administered anonymously , and they require less time 
than interviews. The disadvantages are that questions cannot be clarified in a mail-in 
questionnaire, response rate can be low, generalizations cannot be made unless a high rate 
of return is obtained, respondents may leave questions blank resulting in incomplete data, 
and the responses may represent only the extreme opinions (Worden, 1987). It was 
necessary to utilize questionnaires given the amount of time that the evaluators were in the 
country. The completion of questionnaires allowed for the data to be compiled and 
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analyzed in-country without the direct supervision of the evaluators. The directions for 
completing the questionnaire and the purpose of the study could be explained in a cover 
letter. There was a high response rate with 61 out of 73 (83.6%) surveys returned. 
However, some of the questionnaires were missing a page and others were not filled out 
completely which decreases the number of responses on some items. 

The results of this evaluation were compiled and distributed to the Government of 
Malawi and the USDA. It included the findings from each type of questionnaire, including 
the means on each of the Likert items and recommendations for the next 5 years. The 
objectives of the training as established by the Government of Malawi placed a lot of 
emphasis on the number of training sessions, number of people involved in the training, and 
number of employees who had completed advanced degrees in the U.S. The evaluators 
attempted to construct the evaluation to include not only the quantity of people involved 
in the programs, but also the quality of those programs and their application to agricultural 
development. The surveys relied on the participants' ability to reconstruct their graduate 
education and determine its relevance to their present situation. This reconstruction is one 
of the focuses of ex-post-facto evaluation designs, but also threatens the validity of the 
findings. However, this ex-post-facto evaluation design provides valuable information as to 
the influence of graduate education (the treatment) on the participants' to secure the 
positions that they presently hold (the career outcome). Further analysis of the 
questionnaires given to the long-term, off-shore graduate education participants will be 
presented in a monograph compiled by the Department of Adult and Youth Education at 
Washington State University. This monograph will focus on a comparison of tue data from 
David Acker's original work in Tanzania and these data from Malawi. 
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Implementation 

After developing the theoretical framework and analyzing the background of an 
evaluation that was previously completed, the final step was to apply those principles io the 
evaluation of the pre-departure orientation program. This evaluation was structured using 
the Thirteen Major Steps in Program Evaluation as a reference (Long, 1987). The writer 
utilized the ex-post-facto evaluation design and the program outcome model to construct an 
evaluation of this program. The short, intermediate and long-run outcomes were identified 
and the activities and assessments involved with each of those were examined. The methods 
of data collection, characteristics of the sample, and data analysis techniques were explained. 
The results and limitations of the evaluation were discussed. Finally, ie conclusions and 
recommendations for further study into this area were examined. 

Introduction: 

The rigors of graduate education are well known to graduate students from all 
nations. When you add to those difficulties the adaptation to foreign country and culture, 
the problem is compounded. To decrease the amount of dissonance that the Malawian 
student feels while studying in the U.S., an orientation program was planned for all students 
leaving for study in the U.S. The program was organized by the USAID Training Office and 
the Ministry of Agriculture Training Office. The educational objectives of the orientation 
were: 

• Develop an understanding of USAID, USDA, GOM and University 
regulations, policies and procedures. 

• Increase understanding of how to develop strategies and techniques to 
adjust to American culture. 
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• Increase understanding of how to adjust to the U.S. universities. 

• the academic advisor 

• program of study 

• study habits 

• thesis/research 

• student organizations 

• international student office 

The orientation was a two day session that was conducted within the month prior to leaving. 
The computer training was conducted separately and PIO/Ps were completed during a two- 
day work session. The orientation organizer ranked the program a 4.5 on a scale of 1-5, 
where l=none of the needs met and 5= all needs met (personal interview with Dr. Tom 
Trail, Training Advisor). 

The purpose of this evaluation is to determine the relationship between the 
orientation program and the perceived relevance of the graduate education for those who 
participated in it as compared with those whc did not. Also, to determine if there was any 
difference in the demographic make-up of the two samples. 

Methodology 

The sample is made up of individuals who participated in long-term graduate 
education in the U.S. (n=60). Forty-six individuals participated in the orientation program, 
14 did not. The evaluation design used is the ex-post-facto (Mohr, 1988) and the evaluation 
model used is the program outcome model (Miller, 1979). The program outcomes have 
been identified and an outcome line for the orientation program has been developed (Figure 
4). The instrument that is being used to estimated the relevance of graduate education was 
developed by David Acker (Appendix B) and was utilized in the 1990 evaluation of NRDP 
V for the Government of Malawi conducted by the Department of Adult and Youth 
Education, Washington State University. 
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Figure 4. 
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The dependent variable is the relevance of graduate education to career and to 
national agricultural development. The independent variable is participation in the pre- 
departure orientation. Intervening variables could include demographic characteristics: age, 
and gender; graduate educational characteristics: major, university, level of degree; and 
professional characteristics: position, years to retirement, type of employer. 

Means were calculated for the 28 items which pertain to the perceived relevance of 
graduate education to career development for each sub-sample. T-tests were used to 
determine if any significant difference that existed between the mean relevance values for 
the two groups. Ilie procedure was repeated for the 9 items which pertain to perceived 
relevance of graduate education to national agricultural development. A probability level 
of .05 was used to determine if the means of samples are significantly different. Frequencies 
of response for other demographic variables was calculated, as well as the frequency of 
response for questions 19a and 19b which asked the participants to . ■ .«? the adequacy of 
academic counseling prior to departure (19a) and while studying in the U.S. (19b) (rated on 
the following scale 1= adequate, 2= inadequate, 3= did not receive). 

Threats to Validity 

This evaluation used the ex-post-facto design which encompassed the reconstruction 
of events and decentralized selection. The threats to validity included time-order sequencing 
and spuriousness. It was necessary to use this design because the selection of participants 
for graduate education was decentralized. Several different agencies within the Ministry of 
Agriculture nominated individuals for participation in graduate study and the donor agencies 
selected them. Within the sample, the participants in the orientation program were 
volunteers, which contributed to the decentralized nature of the selection process. Also, 
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individuals were asked to reconstruct events to determine their perceptions of the relevance 
of their graduate education to their career and to national agricultural development. 
Spuriousness was a threat to this study. Due to the lack of pre-test information, it was 
virtually impossible to deterr/une if any changes in relevance were due to the orientation 
program or to other events that could have occurred prior to their leaving their home 
country, while studying in the U.S., or after their return. The subjects received different 
types of treatment for different lengths of time which poses a threat to the external validity 
of the study. The section of Acker's survey that solicited responses regarding the emphasis 
in training with respect to several different areas could provide insight into the differences 
in the treatment experienced by the participants. 

Findings 

Means were calculated for the questions that pertained to relevance of graduate 
education to career for each group. The results are represented in Table 1. There was no 
significant difference between the means of these two sub-groups for this item. For the 
items which pertained to the relevance of national agricultural development, means were 
also calculated and the values are represented in Table 1. There was no significant 
difference between the two sub-groups with respect to national agricultural development. 
Frequencies were calculated on the responses to questions 19a and 19b dealing with the 
adequacy of academic counseling prior to departure and while studying in the U.S. The 
results are presented in Tables 2 and 3. 

Of those who participated in the orientation, 83% were male (n=38) and 17% were 
female (n=8). Seventy-four percent of the participants were going to pursue M Sc. degrees 
and 22% were doctoral students. The average age of the non-participants was 40. 
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Table 1. 



Means for Relevance to Career and 
National Agricultural Development 
(scale: l*very relevant, 5=not relevant) 

Relevance 

Career Agricultural Development 



Participated in 2.00 (n=46) 2.63 (n=46) 
Orientation 

Did not 2.08 (n=14) 2.58 (n=14) 

Participate 

* no significant difference at p<.05. 



Table 2. 



Adequacy of Academic Counseling Prior to 
Departure (Percentage of Responses) 

Participated Adequate Inadequate Did Not No 

in Pre-Departure Receive Response 

Orientation 

Hi Ml N % N % 



Yes 17 37 9 20 13 28 7 15 



No 3 21 3 21 7 50 1 7 
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The remaining 4% indicated that they were bachelor's degree seekers. The average age for 
the orientation participants was 38 years old. In the group that did not participate in 
orientation, 71% were male (n= 10) and 29% were female (n=4). Sixty-four percent of the 
students were doctoral students and 36% were M Sc. 
Table 3. 



Adequacy of Academic Counseling 

While Studying in the U.S. 
(Percentage of Responses) 

Participated Adequate Inadequate Did Not No 

in Pre-Departure Receive Response 

Orientation 

N 1 Hi- N % N % 



Yes 20 43 12 26 6 13 8 17 

No 7 50 5 36 1 7 1 1 
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Conclusions and Discussion 
From the data presented, it can be concluded that there was no difference in the 
perceived relevance of graduate education to career and national development for those 
individuals who participated in a pre-departure orientation as compared to those who did 
not. However, the demographic analysis of the two groups produced an interesting finding, 
the bulk cf individuals who did not participate in the orientation program were doctoral 
students who had previously completed master's level work in the U.S. (9 out or 14). That 
previous experience at a U.S. university could have served the same purpose as the 
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orientation program for the other individuals, This exposure to U.S. graduate education was 
not controlled for in this evaluation. Conversely, most of the participants in the orientation 
program were embarking on master's study in the U.S. after receiving their undergraduate 
degree in Malawi. 

There was a difference in the percentage of individuals who reported adequate pre- 
departure academic counseling between the orientation participants and non-participants. 
Thirty-seven percent of the orientation participants reported adequate pre-departure 
academic counseling. This compares with 21% of the non-participants. Fifty percent of the 
non-participants reported that they did not receive any pre-departure academic counseling 
as compared to 28% of the orientation participants. It is unclear why the number of 
individuals who participated in the orientation program who reported that they did not 
receive any pre-departure academic counseling is so high. Perhaps the wording of the 
question could have influenced the responses or the fact that the respondents were being 
asked to reflect back on something that happened several years ago. This type of 
discrepancy is to be expected with the ex-post-facto design. 

Another issue that must be addressed when focusing on the ex-post-facto evaluation 
of the orientation program is that the questionnaire was not designed specifically to assess 
the outcome of the orientation program. The writer is taking a survey that was 
administered for a different purpose and extrapolating the responses for certain items with 
respect to participation in an orientation program. More conclusive evidence of the 
achievement of program outcomes would possibly result from an instrument designed and 
administered specifically for the purpose of evaluating the orientation program. 
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Limitations and Recommendations for Further Study 
One of the biggest limitations to the generalization of the results observed in the 
Malawi study to a larger population is the fact that the selection was decentralized. There 
arc so many different factors that could have affected the perceptions of relevance by the 
evaluation participants. The lack of a pre-test compounds the problem of establishing a 
baseline for this study. Using the ex-post-facto design and trying to separate the sample 
based on the intervening variables could help to account for their association with the 
outcome. However, experimental or quasi-experimental studies should be conducted in 
which some of the intervening variables are controlled in a more structured setting in order 
to cement any conclusions about their effects on the outcome. 

No decisions regarding the continuation of the orientation program should be made 
based on this assessment. Further research should also be conducted which addresses the 
objectives of the orientation program specifically to determine if similar programs could be 
effective with other graduate students from developing countries who will begin programs 
in the U.S. It would also be interesting to ascertain if the orientation program has 
continued, and if so, has the format changed or do they still incorporate the same 
educational objectives. Perhaps a comparison between countries that provide an orientation 
and those which do not would be helpful in developing some educational objectives and a 
program plan that could be used in many USAID Missions in developing countries across 
Africa or the world. 

A comparison between the perceived relevance of graduate educatic using the 
Acker instrument and the actual amount of knowledge/skills that the participants use in 
their job could provide some interesting results. The actual amount of knowledge gain 
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would have to be assessed if the orientation program has continued, if so has it changed. 
Perhaps a comparison between countries that provide an orientation and those which do not 
would be helpful in developing some educational objectives and a program plan that could 
be used in many USAID Missions in developing countries across Africa or the world. 



30 



References 

Acker, D. G. (1988). Relevance o f U.S. Gradua te Educatio n in Agriculture to Careers in 
Agricultural Development of East African Agriculturalists . Unpublished dissertation, 
Oregon State University, Corvallis, Oregon. 

Gellman, E.S. (1973). Statistics for Teachers . New York: Harper & Row. 

Jimmerson, R. M. (1990) Impact Evaluation Plan for Training Branch NRDP V . Report 
for the Ministry of Agriculture, Malawi, East Africa. 

Jimmerson, R. M. Personal Interview, (April, 1991). 

Long, J. S., & Jimmerson (1990). Impacts of M inistry of Agriculture Staff Training in the 
National Rural Development Programme V (1986-19901 Report for the Ministry of 
Agriculture, Malawi, East Africa. Washington State University, Pullman, 
Washington. 

Long, . S. (1987) Evaluating for Decision Making: A Program Evaluation Reference for 
Washington Cooperative Extension Faculty . Washington State University, Pullman, 
Washington. 

Miller, R. W. (1979). Evaluative Research in Rural Development: Concepts. Methods. 
Issues . Cornell University Press. 

Mohr, L. B. (1988). Impact Analysis for Program Evaluation . Chicago: The Dorsey Press. 

Trail, T. F. Personal Interview (April, 1991). 

Worden, P. E. (1987^ A Source Book for Program Evaluation and Accountability . Colorado 
State University Cooperative Extension, Fort Collins, Colorado. 



36 



